Using Polling Results as Discrete Metrics for Content Quality Prediction Model

ABSTRACT

A social networking system presents content items to users, who then provide feedback regarding pairs of content items. The feedback includes a selection of a content item of the pair of content items that was preferred by the user over the other content item. The social networking system uses this information to train a predictive model that scores content items based on quality. The content items may be advertisements. The social networking system uses the pair-wise comparisons of the advertisements to determine feedback coefficients in an advertising quality score prediction model using regression analysis of the pair-wise comparisons for each predictive factor in the model. In this way, the pair-wise comparisons are used to train the prediction model to understand which advertisements are more enjoyable than others. A feedback coefficient for each predictive factor may be computed based on the preferences received from the group of users.

BACKGROUND

This invention relates generally to social networking, and in particularto training a content quality metric prediction model using pollingresults.

Online advertising has quickly become a major channel through whichadvertisers market their products and services. Traditional performancemetrics are used primarily to determine the effectiveness ofadvertisements, measured for example by click through and conversionrates. These traditional performance metrics do not evaluate the qualityof an advertisement, including the content of the advertisement. Hence,the question of whether users actually enjoyed an advertisement remainsunanswered by these performance metrics.

One problem of measuring and predicting the quality of advertising stemsfrom the inherently subjective nature of audiovisual content andpeople's reactions to it. This is illustrated by the example of a musiccompetition in which three judges rate, on a scale from 1 to 10, theperformance of a singer without being able to judge the singer's visualperformance. In this hypothetical, the judges are tasked withquantitatively scoring the singer based on the singer's vocal abilityand musical performance. Each of the judges may arrive at differentscores due to their relative scoring biases. One judge may be verygenerous in the scoring and only delineate between singers on a muchsmaller scale than another judge who may be very harsh in his scoring.Regardless, an absolute winner of the competition is eventuallydetermined by normalizing the scores. But the judges' scores are onlyapplicable to the specific singers in the competition, not all singersin the world.

Similar to the singing competition hypothetical, advertising models haverelied on the effectiveness of an advertisement, such as asking a focusgroup how likely they are to buy the product mentioned in theadvertisement in the next 6 months, on a scale of 1 to 5. Advertisersand publishers have no way to determine, based on the data gathered,which advertisements are of higher quality and which advertisements areof lower quality. Based on the data gathered, advertisers may onlyextrapolate on a small sample size the subjective opinions of focusgroup members. Advertisers rely on the focus groups to determine if theadvertisement will be effective, not whether the people in the focusgroups actually enjoyed the advertisement.

Attractive advertisements tend to increase engagement with theadvertiser's brand, leading to more user traffic on the publisher'swebsite and an increase in the overall advertisement fees collected bythe online services. Social networking systems have also enabledadvertisers to let users share interesting advertisements with theirconnections on the social networking system, creating “viral”advertising. This “word of mouth” advertising is difficult to generatebecause advertisers and publishers do not have an accurate sense of whatadvertisements are enjoyable, and, consequently, more likely to goviral.

To take advantage of the millions of users that use social networkingsystems, advertisers need better metrics on the content of theiradvertisements. Publishers of advertisements have not created tools ortechniques for advertisers to receive feedback on the quality of theiradvertisements with respect to the user experience of the content withinthe advertisements. Tools and methods are needed to address this problemof determining a discrete measurement of content quality.

SUMMARY

Embodiments of the invention relate to predicting a quality metric forcontent based on user polling. A social networking system presentscontent items to users, who then provide feedback regarding pairs ofcontent items. The feedback includes a selection of a content item ofthe pair of content items that was preferred by the user over the othercontent item. The social networking system uses this user feedbackinformation to train a predictive model that scores content items basedon a prediction of the perceived quality of the content items to a user.

In one embodiment, the content items are advertisements that areprovided by advertisers. The social networking system uses the pair-wisecomparisons of the advertisements to determine feedback coefficients inan advertising quality score prediction model using regression analysisof the pair-wise comparisons for each predictive factor in the model. Inthis way, the pair-wise comparisons are used to train the predictionmodel to understand which advertisements are more enjoyable than others.

In one embodiment, a feedback coefficient for each predictive factor iscomputed based on the preferences received from the group of users. Thefeedback coefficient is computed by estimating the feedback responsefrom similar users based on a statistical model. The statistical modelmay take into account the feedback received from the group of users.

In one embodiment, the social networking system generates a content itemin response to receiving a request for the content item from aparticular user. The content item may be embedded with the selectedadvertisements and then sent to the particular user. The content itemmay also include one or more graphical user elements (e.g., icons,symbols, a string of characters or any visual elements) or anycombination thereof for receiving the feedback response about theselected advertisements from the user.

In one embodiment, feedback coefficients for predictive factors in themodel are computed by logistical regression analysis. A quality scorefor a content item is calculated using a formula that combines feedbackcoefficients multiplied by the probabilities that a user feedback eventis likely to occur along with other probability values generated byother processes. For example, positive or negative feedbackprobabilities may be generated by other processes based on feedbackreceived from users. The feedback probabilities represent the users'expected interest in the advertisement. The feedback coefficientsrepresent the weight given to the value of the advertisement to thesocial networking system. The combination of the feedback coefficientsmultiplied by the feedback probabilities may be linear and non-linear.In one embodiment, quality scores may be computed for content items,such as advertisements, for ranking purposes in presenting the contentitems to users of the social networking system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a conceptual diagram illustrating using polling results topredict content quality metrics, according to one embodiment.

FIG. 1B is a conceptual diagram illustrating pair-wise pollingpreferences, according to one embodiment.

FIG. 2 is a network diagram of a system for using polling results topredict content quality metrics, showing a block diagram of the socialnetworking system, in accordance with an embodiment.

FIG. 3 is a high level block diagram illustrating a polling analysismodule that includes various modules for developing a predictive modelof quality metrics for content items on a social networking system, inaccordance with an embodiment.

FIG. 4 is a flowchart illustrating processes for predicting qualitymeasurements for content items on a social networking system, accordingto one embodiment.

DETAILED DESCRIPTION

A social networking system offers its users the ability to communicateand interact with other users of the social networking system. Usersjoin the social networking system and add connections to a number ofother users to whom they desire to be connected. Users of socialnetworking system can provide information about them, and thisinformation is stored as user profiles for the users. For example, userscan provide their age, gender, geographical location, education history,employment history and the like. The information provided by users maybe used by the social networking system to direct information to theuser. For example, the social networking system may recommend socialgroups, events, and potential friends to a user. The social networkingsystem may also use user profile information to direct content items,such as advertisements, to the user, ensuring that only relevantadvertisements are directed to the user. Relevant advertisements ensurethat advertising spending reaches their intended audiences, rather thanwasting shrinking resources on users that are likely to ignore theadvertisement. Similarly, relevant content items enhance the userexperience on the social networking system, enabling a user to view moreinteresting and relevant content items.

After a content item, such as an advertisement, has been presented to auser of a social networking system, a user may choose to mark thatadvertisement as misleading, offensive, uninteresting, repetitive, orotherwise not relevant. The selection of an “X” on the advertisementprovides user feedback to the social networking system for millions ofadvertisements that are served to users. Using this user feedback,feedback probabilities may be generated to estimate the likelihood thata user may mark an advertisement as misleading, offensive,uninteresting, or otherwise not relevant. Similarly, users may select an“X” on all content items on the social networking system to remove thatcontent item from the stream of content items being displayed to theuser. Such a selection of an “X” on a content item indicates the user'sdisapproving preference about that content item. Conversely, a contentitem may be expressly “liked,” meaning that the user has selected a linkthat indicates the user's approving preference about the content item. Auser may also impliedly signify approval of a content item byinteracting with it, such as re-sharing the content item or commentingon it. A feedback probability estimating the likelihood that a user mayclick on an advertisement may also be generated using the user's pasthistory in clicking on advertisements or particular interest in thetopic of the advertisement, such as a pop music concert.

For advertisements, as with other types of content items, users may bemore or less likely to share the advertisement with their friends andconnections on the social networking system. Recording the history of auser's sharing patterns, including the type of content item such as anadvertisement for an upcoming Britney Spears show, the social networkingsystem can generate a probability value for a user that the user willshare a newly-presented advertisement. These probability values andfeedback probabilities may be obtained and then used to determineoverall quality scores for users, as described for example in U.S.application Ser. No. 12/611,874, entitled “User Feedback-Based Selectionand Prioritizing of Online Advertisements,” filed on Nov. 3, 2009, whichis hereby incorporated by reference.

As used herein, feedback coefficients are numerical values that indicateobjective measurements of quality for the feedback probabilities andother probability values that may be combined to generate an overallquality score. Quality scores may be generated for content items for thepurpose of ranking highly relevant content items for users of a socialnetworking system. Quality scores may also be calculated foradvertisements to rank advertisements that will be shown to users thatare of higher quality, higher relevance, and/or higher value to thesocial networking system and the advertisers (e.g., since higher qualityadvertisements encourage engagement with the social networking system).

FIG. 1A is a high level conceptual diagram illustrating a socialnetworking system 100 for predicting a quality metric for content items,according to one embodiment. A polling module 102 interfaces with polledusers 120 to request a selection of a preference over two pollingcontent objects 104. For example, as illustrated in FIG. 1B, the pollingmodule 102 may ask a polled user 120 a to choose the more enjoyableadvertisement between a Pepsi ad, polling content object 104 a, and aGum ad, polling object 104 b. In one embodiment, the overall userexperience is the dimension by which the polled user 120 a is asked toselect a preference. In another embodiment, a more specific questionrelating to one aspect of the user experience, such as asking “Howengaged were you with the ad?” may be asked by the polling module 102.As shown in FIG. 1B, the polled user 120 a preferred the Pepsi ad overthe Gum ad. This preference is stored as a pair-wise comparison object106 a.

Pair-wise comparison objects 106 are used by a polling analysis module108 to quantify the objective quality of polling content objects 104.The polling analysis module 108 uses regression models to generatefeedback coefficients that are stored as feedback coefficient objects110. Additionally, user profile objects 112, associated with each userof the social networking system 100, are used by the polling analysismodule 108 in order to generate a quality score for polling contentobjects 104 and published content objects 116 that are submitted bypublishing users 124, such as advertisers, using the content publishermodule 114.

Returning to the above example, polled user 120 a had a preference of aPepsi ad over a Gum ad, as illustrated by FIG. 1B. Polled user 120 b,presented with polling content objects 104 c and 104 d, preferred aBritney ad over a Democrats ad, as stored in the pair-wise comparisonobject 106 b. Polled user 120 c disfavored a Target ad, polling contentobject 104 e, over a Giants ad, polling content object 104 f, as storedin the pair-wise comparison object 106 c. It is noted that pollingcontent objects 104 need not be regular advertising banners, but mayalso include interactive games, videos, fan pages, heavy involvement ina group dedicated to a brand, and so on, in one embodiment.

In another embodiment, pair-wise comparisons may be captured byanalyzing declared interests in user profile objects in conjunction withthe frequency of interaction with those interests. For example, if auser declares an interest in Cher and Britney, but only shares contentitems that are related to Britney, the polling module 102 may infer thatthe user has a preference of Britney over Cher and store that preferenceas a pair-wise comparison object 106. In yet another embodiment, apolling module 102 uses such information about users from user profileobjects 112 to select polling content objects 104 that are relevant topolled users 120. Then, pair-wise comparison objects 106 may begenerated after a polled user 120 explicitly chooses one polling contentobject 104 over another.

A content presentation module 118 may use the quality metric, or score,generated by the polling analysis module 108 in adjudicating the overquality, and therefore, potential revenue generator, of a publishedcontent object 116 for ranking and targeting purposes. Targeted users122 receive published content objects 116 from the content presentationmodule 118 as part of the user experience in the social networkingsystem 100. In one embodiment, the content presentation module 118targets advertisements, published content objects 116, to targeted users122 using the quality score generated by the polling analysis module108. In another embodiment, the content presentation module 118 targetsother content items, such as links, photos, video, and applications,that are stored as published content objects 116 and to targeted users122 as ranked by the quality score assigned by the polling analysismodule 108. Ranking and targeting content items based on the qualityscore increases the likelihood that users will remain engaged on thesocial networking system 100, as well as the likelihood that users willshare the high quality content items with their connections on thesocial networking system 100. Higher engagement with the socialnetworking system 100, as well as higher engagement with advertisements,benefits both advertisers and administrators of the social networkingsystem because the advertising dollars have more efficiency in brandpenetration, impressions, and other measurable marketing metrics.

System Architecture

FIG. 2 is a high level block diagram illustrating a system environmentsuitable for predicting quality metrics of content items in a socialnetworking system, in accordance with an embodiment of the invention.The system environment comprises one or more client devices 202, thesocial networking system 100, a network 204, and external websites 214.In alternative configurations, different and/or additional modules canbe included in the system.

The client devices 202 comprise one or more computing devices that canreceive user input and can transmit and receive data via the network204. In one embodiment, the client device 202 is a conventional computersystem executing, for example, a Microsoft Windows-compatible operatingsystem (OS), Apple OS X, and/or a Linux distribution. In anotherembodiment, the client device 202 can be a device having computerfunctionality, such as a personal digital assistant (PDA), mobiletelephone, smart-phone, etc. The client device 202 is configured tocommunicate via network 204. The client device 202 can execute anapplication, for example, a browser application that allows a user ofthe client device 202 to interact with the social networking system 100.In another embodiment, the client device 202 interacts with the socialnetworking system 100 through an application programming interface (API)that runs on the native operating system of the client device 202, suchas iOS 4 and ANDROID.

In one embodiment, the network 204 uses standard communicationstechnologies and/or protocols. Thus, the network 204 can include linksusing technologies such as Ethernet, 802.11, worldwide interoperabilityfor microwave access (WiMAX), 3G, 4G, CDMA, digital subscriber line(DSL), etc. Similarly, the networking protocols used on the network 204can include multiprotocol label switching (MPLS), the transmissioncontrol protocol/Internet protocol (TCP/IP), the User Datagram Protocol(UDP), the hypertext transport protocol (HTTP), the simple mail transferprotocol (SMTP), and the file transfer protocol (FTP). The dataexchanged over the network 204 can be represented using technologiesand/or formats including the hypertext markup language (HTML) and theextensible markup language (XML). In addition, all or some of links canbe encrypted using conventional encryption technologies such as securesockets layer (SSL), transport layer security (TLS), and InternetProtocol security (IPsec).

FIG. 2 contains a block diagram of the social networking system 100. Thesocial networking system 100 includes a user profile store 206, apolling store 208, a published content store 210, a pair-wise comparisonstore 212, a web server 216, a feedback coefficient store 218, a pollingmodule 102, a polling analysis module 108, a content publisher module114 and a content presentation module 118. In other embodiments, thesocial networking system 100 may include additional, fewer, or differentmodules for various applications. Conventional components such asnetwork interfaces, security functions, load balancers, failoverservers, management and network operations consoles, and the like arenot shown so as to not obscure the details of the system.

User account information and other related information for a user arestored in the user profile store 206. The user profile informationstored in user profile store 206 describes the users of the socialnetworking system 100, including biographic, demographic, and othertypes of descriptive information, such as work experience, educationalhistory, gender, hobbies or preferences, location, and the like. Theuser profile may also store other information provided by the user, forexample, images or videos. In certain embodiments, images of users maybe tagged with identification information of users of the socialnetworking system 100 displayed in an image. A user profile store 206maintains profile information about users of the social networkingsystem 100, such as age, gender, interests, geographic location, emailaddresses, credit card information, and other personalized information.The user profile store 206 also maintains references to the actionsstored in the action log 104 and performed on objects in the contentstore 212.

The web server 216 links the social networking system 100 via thenetwork 204 to one or more client devices 202; the web server 216 servesweb pages, as well as other web-related content, such as Java, Flash,XML, and so forth. The web server 216 may provide the functionality ofreceiving and routing messages between the social networking system 100and the client devices 202, for example, instant messages, queuedmessages (e.g., email), text and SMS (short message service) messages,or messages sent using any other suitable messaging technique. The usercan send a request to the web server 216 to upload information, forexample, images or videos that are stored in the published content store210. Additionally, the web server 216 may provide API functionality tosend data directly to native client device operating systems, such asiOS, ANDROID, webOS, and RIM.

The action logger 220 is capable of receiving communications from theweb server 216 about user actions on and/or off the social networkingsystem 100. The action logger 220 populates an action log withinformation about user actions to track them. Such actions may include,for example, adding a connection to the other user, sending a message tothe other user, uploading an image, reading a message from the otheruser, viewing content associated with the other user, attending an eventposted by another user, among others. In addition, a number of actionsdescribed in connection with other objects are directed at particularusers, so these actions are associated with those users as well. Thepolling module 102 may, in one embodiment, use the information in theaction log as recorded by the action logger 220 to generate pair-wisecomparison objects 106 from the frequency of interaction with a certainkeyword, topic, or interest. In one embodiment, the polling module 102,in one embodiment, may use information gathered from external websites214 via the web server 216 over the network 204 in order to generatepair-wise comparison objects 106 stored in the pair-wise comparisonstore 212, measuring the interaction (clicks) on external websites. Forexample, a user who visits the external website 214 for GAP, gap.com,one hundred times a month may have a strong brand affinity for GAP.Contrastingly, if that user only visits CNN.com once a month, apair-wise comparison object 106 may be generated by the polling module102 to indicate that the user prefers content on the external websites214 for GAP over CNN.

Pair-wise comparison objects 106 are generated by the polling module 102and are stored in the pair-wise comparison store 212 in the socialnetworking system 100. Polling content objects 104 are stored in thepolling content store 208 and published content objects 116 are storedin the published content store 210. Feedback coefficient objects 110generated by the polling analysis module 108 are stored in the feedbackcoefficient store 218.

The content presentation module 118 may provide published contentobjects 116 stored in the published content store 210 to client devices202 and/or external websites 214 via the web server 216 over the network204. The content publisher module 114 may receive content items from anexternal website 214 or client devices 202 over the network 204 via theweb server 216.

Developing a Predictive Model to Measure Quality in Content Items Basedon Polling

FIG. 3 illustrates a high level block diagram of the polling analysismodule 108 in further detail, in one embodiment. The polling analysismodule 108 includes an external data gathering module 300, a pair-wisecomparison analysis module 302, a feedback coefficient generating module304, a machine learning module 306, and a quality prediction module 308.These modules may perform in conjunction with each other orindependently to develop a predictive model to measure quality ofcontent items based on polling users of a social networking system 100.

An external data gathering module 300 interfaces with external websitesto gather information about users of the social networking system 100.External websites may be used by the polling module 102 to poll a userregarding a pair of advertisements, for example. Information gatheredfrom the external websites by the external data gathering module 300would be used by the polling analysis module 108 in generating apair-wise comparison object 106, in one embodiment. Additionally, anexternal data gathering module 300 may, in one embodiment, use theaction logger 220 in recording the actions of users on external websitesand synthesizing a pair-wise comparison object 106 based on the actions,such as purchasing a pair of Puma shoes or clicking on an advertisementhosted on an external website for a new Mini Cooper. In anotherembodiment, third parties may help aggregate such information aboutusers of a social networking system 100 that indicates preferencesbetween different types of advertising based on the actions of theusers. As a result, the generation of pair-wise comparison objects 106can be outsourced, in one embodiment.

A pair-wise comparison analysis module 302 develops regression modelsand identifies predictive factors that surface as a result of gatheringpair-wise comparison information. For example, suppose that a polleduser 120 chooses an advertisement for Britney's upcoming concert tourover an advertisement for The Strokes's new album, and similar users tothe polled user 120 also overwhelmingly choose Britney advertisementsover advertisements for Mariah Carey's newest music video. The pair-wisecomparison analysis module 302, using the feedback coefficientgenerating module 304, recognizes predictive factors that indicate ahigher feedback coefficient value. In this example, the similarity ofusers, such as demographics, location, and interests, was a predictivefactor in how the similar users preferred the Britney advertisementsover other music advertisements. Using regression analysis for thatpredictive factor, a feedback coefficient value can be generated usingthe feedback coefficient generating module 304.

The feedback coefficient generating module 304 uses regression models toestimate feedback coefficients for a probability value in a qualityscore formula on the social networking system 100. Each predictivefactor, including the feedback probabilities, and other probabilities,has a separate regression model. The predictive factors, in oneembodiment, include the probability that a user will mark theadvertisement as uninteresting, repetitive, offensive, or otherwiseirrelevant, probability that the user will click on the advertisement,probability that the advertisement is a social advertisement, andprobability that the user will share the advertisement with his or herconnections on the social networking system. Predictive factors may beadded or removed at the discretion of the administrators of the socialnetworking system 100.

The regression model for each predictive factor assigns a feedbackcoefficient value to each of the predictive factors based on thepair-wise comparisons received from the polled users. A logisticregression can be used to calculate the feedback coefficient value foreach of the predictive factors. In one embodiment, the feedbackcoefficient values are customized according to each user of the socialnetworking system. In such an embodiment, the feedback coefficient valuevaries depending on the individual user. In another embodiment, thefeedback coefficient values for the predictive factors are held constantbased on the regression analysis of the pair-wise comparisons receivedfrom the polled users. A machine learning module 306 may also be used tofinalize the feedback coefficient values in the feedback coefficientgenerating module 304.

A quality prediction module 308 determines a quality score thatindicates how engaged a specific user may be with a particular contentitem, such as an advertisement. The quality score of an advertisementfor a specific user would be a combination of the feedback coefficientvalues multiplied by their respective probability values associated withthe specific user. This combination may, in one embodiment, benon-linear, meaning that some probability values may be exponentiallylarger than others as selected by administrators of the socialnetworking system 100. The polling analysis module 108, in oneembodiment, adapts the regression model to include or exclude predictivefactors that are determined to be relevant or not relevant in accuratelypredicting the quality score of content items based on machine learningand heuristics analysis of the targeted users.

Using Pair-Wise Comparisons to Generate Feedback Coefficients to PredictContent Quality

FIG. 4 is a flowchart diagram depicting a process of generating feedbackcoefficients for predicting a quality score of content items to bepresented to users of a social networking system, in accordance with anembodiment of the invention. Pair-wise comparisons indicating apreference for a first content item over a second content item arereceived 400 from a group of users. Objects associated with the firstand second content items and the users are modified 402 to indicate thepair-wise comparisons.

At least one predictive factor associated with the preference isidentified 404. The predictive factors may include any signals ofinformation available in the social networking system, such as staticdata (e.g., information in a user's profile, such as interests), socialdata (e.g., information about the user's connections), historical data(e.g., information about the user's previous interactions with contentitems), and any other information signal that may be helpful inpredicting a user's impression of a content item's quality. Theidentified predictive factors are analyzed 406 using a regression model,which is trained using the results of the pair-wise comparisons 406. Afeedback coefficient associated with each identified predictive factoris adjusted 408 or otherwise obtained based on this training In oneembodiment, an initial feedback coefficient is predetermined byadministrators of the social networking system 100. As more feedback isreceived from polled users 120, the pair-wise comparisons can beincorporated into the feedback coefficients using this process. Finally,if other predictive factors associated with the pair-wise comparisonshave been identified, then the process repeats at step 406. Otherwise,the feedback coefficients for the quality metric formulas associatedwith the pair-wise comparison are stored 410. Once the model isobtained, it may then be used to predict other users' impressions of thequality of selected content items.

SUMMARY

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

For example, although embodiments of the invention have been describedin terms of advertisements, pair-wise comparison polling can beimplemented to rank other content items besides advertisements. Forexample, news articles that are shared between users may be ranked inpresentation to other users using the above-described method todetermine an objective measure of the quality of the news articles asdetermined from polling results.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

1. A computer-implemented method for predicting quality of content itemson a social networking system, the method comprising: receiving aplurality of pair-wise comparisons from polled users of the socialnetworking system, each pair-wise comparison indicating a preference ofa first content item over a second content item; modifying contentobjects associated with the first and second content items to indicatethe plurality of pair-wise comparisons; modifying user profile objectsassociated with the polled users of the social networking system toindicate the preference; identifying at least one predictive factorassociated with the preference; for each identified predictive factor,determining a feedback coefficient for the identified predictive factorbased on the plurality of pair-wise comparisons; and storing thefeedback coefficients for the identified predictive factors in acomputer-readable storage medium.
 2. The computer-implemented method ofclaim 1, wherein determining a feedback coefficient for the identifiedcontent quality predictive factor based on the plurality of pair-wisecomparisons comprises: analyzing the plurality of pair-wise comparisons,the comparisons indicating preferences, using a logistic regression; anddetermining a feedback coefficient for the identified content qualitypredictive factor as a result of the analyzing.
 3. Thecomputer-implemented method of claim 1, further comprising: receivingnew content items for display to a user; generating a quality score foreach of the new content items based on a combination of the storedfeedback coefficients for the identified content quality predictivefactors and profile information about the user; and ranking the newcontent items for display based on the generated quality scores.
 4. Thecomputer-implemented method of claim 1, wherein the content itemsinclude advertisements for display to users of the social networkingsystem.
 5. The computer-implemented method of claim 1, wherein thecontent items include links to external websites shared by users of thesocial networking system.
 6. The computer-implemented method of claim 1,wherein the content items include photos shared by users of the socialnetworking system.
 7. The computer-implemented method of claim 1,wherein the content items include music shared by users of the socialnetworking system.
 8. The computer-implemented method of claim 1,wherein the content items include videos shared by users of the socialnetworking system.
 9. The computer-implemented method of claim 1,wherein the content items include specified users shared by users of thesocial networking system.
 10. The computer-implemented method of claim1, wherein the content items include fan pages shared by users of thesocial networking system.
 11. The computer-implemented method of claim1, wherein the content items include groups of users of the socialnetworking system.
 12. The computer-implemented method of claim 1,wherein the content items include interests shared by users of thesocial networking system.
 13. The computer-implemented method of claim1, wherein receiving a plurality of pair-wise comparisons from polledusers of the social networking system, each pair-wise comparisonindicating a preference of a first content item over a second contentitem comprises: receiving selections of links associated with indicatingthe preference of the first content item over the second content item.14. The computer-implemented method of claim 1, wherein receiving aplurality of pair-wise comparisons from polled users of the socialnetworking system, each pair-wise comparison indicating a preference ofa first content item over a second content item comprises: receivingquantifiable scores of a plurality of content items; and for each of theplurality of content items, generating a unique pair-wise comparison oftwo of the plurality of content items by ranking the content items byquantifiable scores associated with the content items, the rankingindicating the preference of the first content item over the secondcontent item.
 15. A computer-implemented method for providing qualityadvertisements to users of a social networking system, the methodcomprising: receiving a plurality of pair-wise comparisons from polledusers of the social networking system, each pair-wise comparisonindicating a preference of a first advertisement over a secondadvertisement; identifying at least one predictive factor associatedwith the preference; for each identified predictive factor, determininga feedback coefficient for the identified predictive factor based on theplurality of pair-wise comparisons; for each advertisement available fordisplay to a user, determining a quality perception score for the userby combining the feedback coefficients and profile information about theuser retrieved from a user profile object associated with the user;providing for display to the user a quality advertisement based on thequality perception score associated with the quality advertisement. 16.A computer-readable storage medium storing instructions, theinstructions when executed by a processor in a social networking systemfor predicting quality of content items, causes the processor to:receive from a group of users of the social networking system aplurality of pair-wise comparisons, each pair-wise comparison indicatinga preference of a first content item over a second content item; modifycontent objects associated with the first and second content items toindicate the plurality of pair-wise comparisons; modify user profileobjects associated with the group of users to indicate the preference;identify at least one content quality predictive factor associated withthe preference; for each identified content quality predictive factor,determine a feedback coefficient for the identified content qualitypredictive factor based on the plurality of pair-wise comparisons; storethe feedback coefficients for the identified content quality predictivefactors in the computer-readable storage medium.